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To make training cost-effective fcr ite small systens 


customers, Digital Equipment Corporation (DEC) is providing 
self-instructional packages that can be used by custosers on-site for 
training peogle to run their systems. To control the quality of this 
training, Ccoaputer Hanaged Instruction (CMI) is ncw incorporated into 
ecrme of these packages. This CSI component uses a sequential 
probability algoritha that allows tests to vary in length depending 
upon the learner’s skill level. This algorithm assures that the test 
results are statistically reliable while keering the length of the 
tests as short as possible. All of the prograss that sake up this CMI 
system are written in BASIC; all of the CHI frograms and data files 
for about 800 tests items will fit on a single, dual-density 
diskette. These characteristics sake the CHI system applicable to 
internal and large systeas training as well as saall systeas — 
training, because it is small enough to fit on a diskette yet 
sophisticated enough to handle more and larger item banks if 
additional disk space is available. (Author/VT) 
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The cost of small computer systems is constantly decreas- 
ing, but the cost of training people to run these systems 
is constantly increasing. To make training cost-effective 
fow its small systems customers, Digital Equipment Corpora- 
tion has been providing self-instructional. packages for 


» S@veral years that can be used by our customers on-site. 
iTo control the quality of this training, Digital is now in- 
* corporating Computer-Managed Instruction (CMI) into some of 


these packages. THis CMI component uses a sequential pro- 
bability algorithm that allows tests to vary in length de- 
pending upon the learner's skill level. This algorithm as- 
sures that the tests results are statistically reliable 
while keeping the length of the tests as short as possible. 
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CUSTOMER TRAINING AT DIGITAL 


x 


Digital offered its first customer training courses in lec- 
ture format in the basement of its legendary "mill" in 
1962. Today, Digital's Educational Services Department 
offers over 100 lecture and self-instructional customer 
courses that are available in six regional training centers 
in the United States and eleven centers in other parts of 
the globe. 

In‘ 1975, Digital began to market very small, turn-key sys- 
tems. The costs of these small systems are so low that 
even with our regional training centers it is often not fi- 
.nancially feasible for customers to send their employees to 
Digital for training. 


To make cost-effective training available to these. cus- 
tomers, we have developed a new form of training called 
"tutorial documentation". This documentation is designed 
specifically for teaching completely inexperienced users to 
operate their systems without having a Digital instructor 
present. 


Traditional customer education at Digital continues to grow 

with the sales of our medium- and large-scale systems, and - 
tutorial documentation has now added a new dimension to the 
“training services that we can offer our small systems cus- 

tomers. We have found, however, that the use of tutorial 
documentation at customer sites presents two new problems — . 
of its own: 


e it is difficult to control the use of these 
materials when they are used on customer sites 
as well as we can when they are used in Digi- 
tal facilities, and 


e it is difficult to get accurate feedback on 
the strengths’ and weaknesses of anes materi- 
als from our customers. 


Digital's Tutorial Documentation Group is addressing these 
problems’ by writing computer-managed instructional materi- 
als to run on some of these small systems. These CMI ma- 
terials use the customers' computers themselves to control 
their learning and collect data that we can use to assess 
the effectiveness of the tutorial documentation. 


The remainder of this paper describes an existing 
computer-managed instruction program that is coupled with 


the tutorial documentation for one of Digital's new small 
wv Systems. 
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CMI IN A CUSTOMER ENVIRONMENT 


Interaction with Tutorial Documentation 


Digital's tutorial documentation is written in a modular- 
ized format. (The modules are arranged in a hierarchy, 
based on the prerequisite relationships of their objec- 
tives. Each module contains a list of its objectives, text 
and ‘diagrams to help learners master these objectives, and 
exercises to be performed both in the workbook and on-line. 
The\ test item banks for the module tests are all _ stored 
on-line. : 
| / 

Before learners begin work on the tutorial documentation, 


they| take the pretest for the first module (see Figureg 1). 
se tb 3 they can demonstrate mastery on this test, they/ are 
branched to the pretest for the next module in the hjerar- 


Ba This loop continues until the learners. come to/a test 

which they cannot demonstrate mastery. At this point, 
Ebey are directed to study that module off-line, and return 
to the CMI system when they are ready for the posttest. 


An LPP ESSAY quality of the CMI approach is that .it gets 
users |on-line as soon as possible and therefore has a de- 
finite| Hawthorne Effect. In the past, customers often just 
skipped the tests that were included in our training pack- 
ages, because they felt -- erroneously -- that testing be- 
nefits only the teacher. It is difficult, if not impossi- 
ble, to change this feeling, but we can capitalize on the 
Hawthorne Effect to get more of our customers to take the 
tests. |The directions for booting the system and _ running 


the CMI registration program are provided in the tutorial 


documentation in cookbook terms, so: that even the most 
inexperienced of our users can get the CMI system on the 
air. 


1 
General Characteristics 


The, CMI registration program allows users to register them- 
selves interactively. It records their first and last 
names (making sure that each is unique), and their ad- 
dresses. This program also allows users to view the status 
of their work on each of the modules in the course. 

The CMI test administration program presents true/false and 
multiple choice items (with either four or five alterna- 


.tives) to users on a CRT terminal. These tests are gener- 


ated interactively in real time. The items are randomly 
selected from item banks that are categorized by module and 
objective. A typical item display is shown in Figure 2. 
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Take PREtest 
for Module 1° 


Take POSTtest 
for next module 


~ All 
modules 
completed? 


. Master? 


Study module and 
do exercises 


Take POSTtest 


| 
Figure 1 


INTERACTION BETWEEN ON-LINE CMI SYSTEM AND 
OFF-LINE TUTORIAL DOCUMENTATION 


| 
| 
237 5 | 


lle 


SMALL SYSTEMS CMI pa Wea 
Jesse M. Heines ee 


4 
“ x ‘ bs 


MULTIPLE CHOICE.. ‘Enter the letter of the alternative that BEST 
answers the question or’ completes. the ‘sentence in the item below. 


Enter’ SKIP if you don't know ‘the answer (counted. as incorrect). 
Enter QUIT if yOu must terminate this test before -it is completed. 


Press the RETURN key after you type each entry. 


10. Which of the following statements will cause 


196 


4 


to be displayed onethe terfninal? 


PRINT "102 + 94" 

PRINT 102 + 94 . 
PRINT ¥102 + 94 = 102494" 
PRINT "102 + 94" = 102494 


Your choice? 


Figure 2 


SAMPLE DISPLAY OF A MULTIPLE CHOICE ITEM 
PRESENTED TO THE STUDENT 
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The system is highly human-engineered to make it as 
fool-proof as possible in'a customer environment. For ex- 
ample, it provides the options "SKIP" and "QUIT" as_ shown 
in Figure 2, and it provides "error" messages in plain En- 
glish: if the user enters "g" in response to a true/false 
item, the system will print, "Please enter only T, F, SKIP, 
or QUIT". It will then erase the user's previous’ response 
and make room for him or her to enter another one. 


THE MASTERY DECISION MODEL 


§ 


Sequential Testing 


Even-with a considerable Hawthorne Effect, customers: still 
do not like to be tested. (There is always someone in 
every customer training course who will’ say, "I paid’ my 
money to be taught, not tested!") It is therefore impor- 
tant to keep the length of the module tests: as. short as 
possible.: It is-useless, however, to make these tests so 
_ Short that their reliability approaches zero. 


Since 1974, 411 courses developed by Digital's Educational ' 
Services ‘Department have been developed using .a criterion- 
referenced philosophy. This philosophy is especially ap- 
plicable to industrial training, because we. are interested 
in. individual performance rather than a compari'son between 
_learners. We ‘therefore required the CMI: system to apply 
this philosophy as well. In 1975, this author reviewed the 
literature on criterion-referenced and computer-assisted 
testing, and the reader is referred to ERIC Document | 
ED116633 for ‘the complete text of that report. 


The most highly "developed criterion-referenced decision 


, model -that takes advantage of the capabilities of interac- 


tive! computing is one developed by Richard Férguson [1]. 
Ferguson's model is based on Wald's sequential probability 
test ratio [2]. This model allows two criterion scores to 
. be defined, PO and P11. Both of these scores are expressed 
in terms of percentages of. correct responses. Learners 
whose scores are greater than PO are classified as masters, 
and fearners whose scores are less than P1 are classified 
as non-masters. Learners whose. scores fall between PO and 
P1 are pr Seanved ‘with another item. 


{1] Ferguson, Richard L. . Computer assistance for indivi- 
dualized measurement. Learning Resource and Develop- 
ment Center, University of Pittsburgh. March, 1971. 

\ 


[2] Wald, Abraham, Sequential] .Anaylsis. John ‘Wiley & 
: Sons,. Inc., New York, ‘ - ‘ 
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This model also takes into account the probability with 
which the test developer is wilting to. allow Type I and 
Type II errors to occur (3]. For the purposes of the fol- 
lowing discussion, let us define "A" as the’ probability — 
- with which we are willing to allow a Type I error to occur, 
and "B" as the probability with which we are willing to 
allow a Type II error to occur. -The test developer’ can 
then assign values to PO, P1, A, and B to determine the 
learners' mastery state to. any desired degree of accuracy. 


Computations 


After each test item is administered, the student's * score, 
"S", is computed using the formula: 


S = C x log(P1/P0) + Wx log((1-P1)/(1-P0) ) 


where "C" is the number of items answered correctly, and 
"Ww" is the number of items answered incorrectly. 


If the student is a master, 
S ¢ log(B/(1-A)) 

If the student is a non-master, 
S > log((1-B)/A) 


If neither of these inequalities is true, i.e., if 


log(B/(1-A)) < S <¢ log((1-B)/A) 


another test item is administered. The system continues in 
this manner until one of the first two inequalities is’ true 
or until 30 items have been administered. If no decision 
can be made after 30 items, the system classifies the stu- 
dent based on the differences between his or her score and 
the two criteria. The student is classified in the cate- 


gory whose criterion is closest to his or her’ computed 
score. 


(3) A Type I error is a false positive error and _ occurs 
when a true non-master is classified as a master by 
the test. A Type II error is a false negative error 
and .occurs when a true master is classified as a’ 
non-master. 
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IMPLEMENTATION 


* 


Test Parameters 


As mentioned previously, our CMI system generates both pre- 
tests _and posttests. For this reason, it is important to 
realize that .the serioushess of making Type I and Type II 
errors is different on pretests and posttests. If we make 
a Type I error on a pretest, we will be telling a _ student 
who has. not studied that module to skip instruction that he 
or she really needs. This same error on a posttest.is not 
as serious, because the student will have already studied 
the module at léast once. A Type II error is never as ser- 
ious .as a Type I error, because this situation simply asks 


a student to repeat instruction that he or she does not re- 
ally need. 


To take the relative importance of these errors into consi- 
~ ,deration, our CMI system uses the parameters shown in Table 
1 for the sequential testing decision model. 


Table 1 


SEQUENTIAL TESTING PARAMETERS FOR 
PRETESTS AND POSTTESTS 


For For 
Parameter PREtests POSTtests 
Mastery Criterion (™PO*) O.40 ~ O.85 
Non-Mastery Criterion ("P1") 0.70 0.60 
Allowable Probability of a. 
Type I Error ("A") - 0.025 0.050 
Allowable Probability of a 
Type II EEnor CNBr) 0.06 0.08 


Notice that the allowable probability of a Type I error. on 
a ‘posttest is twice as high as that on a pretest. To see 
how ese parameters affect the mastery decision model in 
terms of raw scores, refer to Figures 3a and 3b. 


Figure 3a shows a graph of the pretest decision rules, 
while Figure’ 3b shows a= graph of the posttest decision . 
rules. Note the difference in the sizes of the two master 
areas and the specific points that are labelled. The point 
labelled "(3,0)" in both graphs indicates,that the earliest 
. that a non-master decision could be made on either test is 
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GRAPHS OF THE PRETEST AND POSTTEST DECISION RULES 


4 


Figure 3a. 


PRETEST 
DECISION 
RULES 


(30,30) 


(30, 25) 


Figure 3d. 


POSTTEST 
DECISION 
RULES 


Figures 3a and 3b 
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after 3 items have been presented. If, at this time, the 


student 


has not answered any items correctly, he or she is 


classified as a nonemaster. 


In Figure 3a, the point labelled "(15,15)" indicates that 


the 


earliest that a master decision could be made on a 


retest is after 15 items have been presented. and all 15 
ave been answered correctly. Contrast this point with the 
one labelled "(9,9)" in Figure 3b. The latter indicates 
that the earliest that a master decision could be made on a 
posttest is after 9 items have been presented and answered 


correctly. 


Therefore, the posttest mastery criterion is 


less stringent than the pretest mastery criterion. This is 


exactly 


what we wanted, because we have already shown that 


an erroneous master decision on a posttest is less’ serious 
than the corresponding error on a pretest [4]. 


Closing the Feedback Loop 


The system on which these CMI programs are currently imple- 
mented is a diskette-based system. The complete CMI system 
will be supplied to users on a diskette even when they pur- 


chase 


minimum configuration. After they complete all 


of the modules in the course, users will be instructed to 


mail their 


diskettes back to Digital in special business 


reply envelopes that will also-be supplied. We will read 


their 


zero the log files if so requested, and return 


the disketbes. The data stored by the programs will allow 
us to do complete criterion-referenced item analysis on the 


users' 
module. 


responses and check the status of the users on each 
Users who complete the entire course will receive 


a diploma when their diskettes are returned. 


m_aeeeeweee = ee eawewe = 


The actual decision model implemented on our CMI: sys- 


slightly more complex than the one discussed 


here, because Wald's sequential probability test ratio 
developed for events that occur with equal proba- 
bilities. Sinee the probability of getting a 
true/false item correct by guessing is different from 
that for a multiple choice item, our system computes a 
weighting factor to correct for the difference between 
our application of Wald's formulas and the probability 
model for which they were originally developed. 
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“FUTURE ENHANCEMENTS 


All of the programs that make up this CMI system are writ- 
ten in the BASIC language. This makes them highly tran- 
sportable to almost all of Digital's operating systems, and 
we plan to use them for most of our tutorial documentation 
projects in the future. In addition, all of the CMI _ pro- 
grams and data files for about 800 test items will fit on a 
single, dual-density diskette (approximately 250K PDP-11 
words). 


These characteristics make the CMI system’ applicable to 
internal .and large systems training as well as small sys- 
tems training, because it is small enough to fit on a disk- 
ette yet. sophisticated enough to handle more>and larger 
item banks if additional disk space is available. 


We hope to begin putting appropriate parts of the actual 
instruction on-line on our small systems in the near future 
to complement the testing and course management functions 
that we have available today. 


